The present disclosure generally relates to performance testing, and more particularly relates to adapting workloads for use in a performance testing tool.
Performance testing is a complex and time-consuming practice. Every aspect of the design, code, and execution environment of software is influenced by performance. Its pervasive nature makes it a critical dimension of quality, especially at enterprise levels, as it plays a central role in software usability. The latest trends in information technology (such as Cloud computing and Service Oriented Architecture) have added to its complexity.
An example of the complexity of performance testing can be readily appreciated in the context of mail applications such as Gmail™ by Google Inc. Gmail™ involves over 100 transactions—creating emails, reading emails, deleting emails, creating calendar events, deleting calendar events, to name a few. Some transactions include no attachments, some include small, medium, large, very large attachments. Typically, a performance engineer models a workload script to emulate expected behaviors that would likely occur in a production environment. The workload script ideally amplifies the path most frequently traversed (as seen in production runs). For example, folks tend to read more emails that they write emails, and they tend to delete some emails and archive others and forward a few others. Some replied-to emails will be reply-to-all while others will be simply reply-to-author, and other transaction possibilities.
Performance test tools monitor the application performance metrics tied to quality of service objectives, such as average response time, under peak workload conditions. The performance test software observes the behavior of complex workloads, such as the volume of transactions processed by an application per second (tps), requests per second, and page loads per second. Due to the complexity of such workloads, a reliable and replicable deployment is not a simple task.
Performance issues can materialize into serious problems, such as outages in production environments. Assessing these issues is particularly difficult, especially since the problems are often related to the test workloads. These problems are known as workload-dependent performance bottlenecks (WDPBs). WDPBs are commonly caused by workload-dependent loops which contain one or more relatively high-consuming operations, such as file Input/Output (I/O) operations, object creation operations, object destruction operations, and user interface updates.
Techniques exist to identify WDPBs, such as current performance testing tools and performance profiling approaches (e.g., call-tree profiling or stack sampling). However these traditional techniques are inefficient because they rely on static workloads. The performance tester must estimate the size of a workload sufficient to identify performance issues, and that estimated workload does not change throughout a test run, which may span several days. Testers often use “standard” workloads, which are adequate for exposing WDPBs in some functions. However, these standard workloads prove insufficient in some cases to identify WDPBs on small or even relatively large workloads. Moreover, it is often unclear to performance testers how large a workload is large enough for a workload to expose any possible WDPBs.
Incorrect assumption of workloads brings to light two inefficiencies in the current performance testing discipline. First, incorrect workload assumption increases the complexity of performance testing and analysis, indirectly increasing the cost and time required for these activities (commonly limited due to project budget or time constraints). Second, this problem also increases the risk of potentially overlooking performance issues which might have serious business consequences, such as unavailability of services or monetary costs.
IBM® Rational Performance Tester® (RPT) is a leading load testing tool in the industry and works as follows. If a tester wants to use RPT for performance testing of an application (e.g., an IBM Portal® environment), the tester first creates a test script with all the transactions that are of interest (e.g., login, search, and logout) and then specifies a workload (e.g. 200 users) which will be used during the entire testing run. The users for tests of this type are commonly virtual users. Virtual users simulate system activities of an actual user. The available workload pool, often subject to licensing thresholds, or system limits, is a delimiting factor.
However, the workload required to identify performance issues might vary dependent on the application. The workload of 200 users might be insufficient if the same tester needs to evaluate the performance of the IBM Lotus Connections® or the IBM WebSphere® (software suites). Different usage scenarios may require different workloads. As an example, it makes sense to test the commonly-used login/logout functionality with a larger workload than, say a rarely-used functionality.